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8.1 INTRODUCTION 


Human interactions still play a significant role despite the advancements in the 
manufacturing landscape with Industry 4.0 technologies. Moreover, the aim of 
Industry 4.0 is not to replace machines with humans but to enhance human capa- 
bility with the support of advanced technologies. The market trends suggest an 
increased demand for customized products, increasing the product variants and 
hence more demand on the shopfloor (Kolla et al., 2020). Therefore, operators need 
systems that can assist them in completing their tasks quicker, more precisely, and 
with a lesser workload. Extended reality (XR) technologies can be employed in the 
industry to give operators an enhanced perception of the instructions in a visual and 
interactive manner (Doolani et al., 2020; Fast-Berglund et al., 2018). The XR 
spectrum includes technologies such as augmented reality (AR), mixed reality 
(MR), and virtual reality (VR). Figure 8.1 illustrates the XR spectrum, which en- 
compasses AR, MR, and VR technologies. 

The virtual extreme of the XR continuum is VR. In VR, the user is completely 
immersed in the virtual world and isolated from the real environment (Mujber et al., 
2004). In a VR environment, the user is surrounded by virtual objects and is unaware 
of reality. On the other hand, the real world is the other extreme, consisting ex- 
clusively of physical objects and free of any virtual objects to improve the operator’s 
perception. AR and MR enhance the user experience by improving reality with virtual 
elements. In an AR environment, the virtual information is overlaid in the natural 
environment. However, in an MR environment, it is possible to develop interactions 
between real and virtual objects. Therefore, MR can be considered an umbrella term 
for AR and VR. Ina nutshell, AR and MR can be differentiated based on the extent of 
interactions between real and virtual elements. 

Numerous advantages of XR technology include traceability, increased intuitive 
learning, decreased error rates, and reliability. AR and MR can help shop floor 
employees in their daily jobs by enabling digital poka-yoke systems and reducing 
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FIGURE 8.1 Extended reality (XR) spectrum illustrating both virtual and real extremes of 
awareness. 


manual errors and rework of complex and unfamiliar assembly tasks. VR can be 
effectively employed in training activities in any manufacturing enterprise. A new 
way of interacting with machines and the shop floor environment is also introduced 
by XR, opening the door for future solutions for human-machine interfaces (HMI). 

This chapter is structured as follows: Section 8.2 presents basic components and 
different types of AR systems, followed by various interaction modalities in 
AR systems in Section 8.3. AR systems architectures and case studies are detailed in 
Section 8.4. This is followed by a brief introduction to VR technology in Section 8.5. 
The important applications of VR technology in manufacturing are presented in 
Section 8.6 and the case studies are highlighted in Section 8.7. A conceptual framework 
of MR is shown in Section 8.8, followed by the conclusion and discussion in 
Section 8.9. 


8.2 BASIC COMPONENTS AND TYPES OF AR SYSTEMS 


With the support of several existing technologies, AR overlays virtual information in 
the real world. As a result, AR improves the operator’s perception of Reality that has 
been supplemented with digital data. The way operators interact with their environ- 
ment and carry out their daily activities will shift because of this paradigm shift. 
There are five essential components in an AR system: capturing technology 
(CT), visualization technology (VT), processing unit (PU), tracking system, and 
user interface (UI). CT includes a camera or a sensor to capture the real world. VTs 
are the essential components of an AR system as they superimpose virtual elements 
in the real world. Example VTs include a mobile phone or a HoloLens®. A PU in an 
AR system analyses the data input and aids in visual rendering. A tracking system 
includes a marker, such as a QR code, used to orient the virtual information 
in relation to the real environment. Some AR systems can use advanced approa- 
ches such as spatial mapping and bypass a tracking system. AR systems with 
markers are often utilized in the industry in contrast with marker-less technologies, 
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FIGURE 8.2 Basic components of an AR system. 


as markers could significantly reduce the computing power in rendering virtual 
information. Finally, the user interacts with the AR system supported by a UI 
(Fraga-Lamas et al., 2018; Lee et al., 2017; Wang et al., 2016). Figure 8.2. illus- 
trates the basic components of an AR system. 


8.2.1 Types of AR Devices 


Three AR devices are used: handheld devices (HHDs), head-mounted devices 
(HMDs), and spatial AR devices. HHDs are devices that a user can manipulate using 
their hands. The most common example of HHDs is mobile phones and tablets. These 
devices employ all the necessary components, such as a camera, display unit, pro- 
cessing unit, and other additional sensors. HMD is a device to wear on the head of 
the user, and these days they are available as part of a helmet. While using the device, 
the user is not entirely immersed in the virtual world as they can see both virtual and 
real environments intact. HHDs employ image sensors, depth cameras, holographic 
optical elements, holographic processing units, and microphones to support the 
development of AR applications. Spatial AR devices display virtual information in a 
real environment using video projectors, beamers, and lights. Currently, logistics and 
warehouse optimization projects use spatial AR technology extensively. 


8.2.2 BENEFITS AND LIMITATIONS OF AR SYSTEMS 


In general, HMDs are favored for assembly applications because they give the user 
a hands-free experience and enhance the user’s perception of the real world because 
the view through an HMD is nearly identical to the real world. On the other hand, 
HHDs are used in everyday life (e.g., tablets or a smartphone). Using an HHD 
lessens the need for training and enables an operator to use the system directly with 
little to no training. HHDs are superior to HMDs from an ergonomic and financial 
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standpoint. In spatial displays, the operator’s hands are free, and they do not need to 
carry anything while performing their tasks. AR systems with markers are less 
computationally intensive compared to marker-less AR systems. Because AR ap- 
plications do not require a lot of processing power, both mid-range and high-end 
devices can be used. Another benefit is portability, which makes it simple to move 
AR applications and hardware from one place to another. 

The limitations of AR systems are associated with the maturity of the available 
technology. HMDs are uncomfortable to wear and have a limited field of view 
(FoV). Limited FoV compromises the operators’ safety and can lead to workplace 
accidents (Palmarini et al., 2018). In some cases, inadequate resolution of the virtual 
information can cause the operator’s sickness when exposed for extended periods. 
HMDs are heavy to wear and work for a prolonged period. Furthermore, HHDs are 
not portable because they require physical support to be held in place. A further 
drawback of an HHD system is the limited size of its UI, which restricts the amount 
of information that can be displayed on it. AR systems that use markers need special 
care concerning the maintenance of the markers. Marker-less AR systems require 
more computational power for object tracking in real-time. 


8.3 INTERACTION MODALITIES IN AR SYSTEMS 


Interaction modality is a way of sensing such as touch (haptic), voice (acoustical or 
auditive), vision (optical), etc. In AR, interaction modalities help users to interact 
with AR interfaces (Esteves et al., 2017). Currently, there are three ways to interact 
with AR systems: touch, touchless, and wearables. These interaction modalities are 
divided into input modalities and output modalities. Input modalities are for the user 
to give instructions to the AR system, and output modalities are for the user to 
receive feedback from AR systems. The AR modalities are shown in Figure 8.3. 

Touch modality is not new in the manufacturing industry. Most of the existing 
operator panels on the mechanical systems have touch modality enabled, either with 
buttons or with touch screen displays. 

With the development of AR technology, especially with HMDs, a new type of 
deictic gesture to interact with AR devices is possible (Williams et al., 2019). These 
gestures are mostly given mid-air with the operator’s hands (Mistry & Maes, 2009). 
For example, raising an index finger in the FoV of smart glass can indicate that the 
operator is ready for a selection, and pinching the index finger with the thumb can 
follow the selection of a particular target. 

Voice, a natural interaction modality, has been the most widely used in our daily 
communication. Yet, this modality is intercepted by machine noises in an industrial 
environment, especially on shop floors. The voice interaction modality enabled 
by speech recognition techniques allows the user to interact with the AR systems 
using voice commands (Nizam et al., 2018). Eye tracking (Liu et al., 2019) is a 
recent development in AR that is derived from complex algorithms that estimate the 
movement of the cornea and pupil of the eye in space. In an HMD powered by eye 
tracking interaction modality, cameras that operate at a high frame rate capture the 
images of the eyes. As the name suggests, screen dwelling (Qian et al., 2020) makes 
interaction with a system possible by gazing at the target for a specified amount 
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FIGURE 8.3 Interaction modalities in AR systems. 


of time. The gyroscope and motion sensors of the system facilitate the dwelling 
interaction. Of course, dwelling is only possible by combining it with another 
modality, such as head, hand, or eye tracking. 

Even though wearables found their way into the fitness industry and health 
tech, they aren’t matured to be used along with AR devices in an industrial en- 
vironment. However, research on wearables such as smart gloves (Hsieh et al., 
2016), belts (Dobbelstein et al., 2015), and wristbands (Hu et al., 2020) is rapidly 
progressing for better-enhanced interactions with smart devices. This chapter 
presents two case studies where touch, gesture, and voice interaction modalities 
are employed in the AR application. 


8.4 AR CASE STUDIES 


All-in-one AR solutions are readily available in the market to meet a wide range of 
industrial needs. These solutions are expensive and, to an extent, overkill for smaller 
applications in manufacturing enterprises. In this section, the authors demonstrated 
the development of AR applications from scratch using inexpensive hardware, open- 
source software, and plugins. The workflow process presented in this section is 
attractive to the needs of SMEs as the capital expenditure required is minimal com- 
pared to off-the-shelf solutions on the market. 
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Both case studies solve the same problem of digitalizing the work instructions to 
assemble a planetary gearbox using two alternative hardware setups and features on 
each. In the first case study, an affordable Android system is used to develop an AR 
solution; in the second, a HoloLens® is proposed with expanded capabilities. 

In terms of software, Unity® is the primary development engine used to create 
interactive 3D experiences. Besides Unity”, C# programming is used for interac- 
tions in AR applications, Vuforia® for marker tracking and orientation of virtual 
elements, and Autodesk® inventor for 3D modeling. Various open-source plugins 
are used from the Unity® store for specific operations in developing AR applica- 
tions. For example, the lean touch plugin from the Unity® store enabled the touch 
modality in the Android AR application. 


8.4.1 Case STUDY PROBLEM 


A planetary gearbox is designed by Autodesk® inventor, which contains a diversity 
of components and requires a number of steps to be assembled. The case studies 
need to simulate the real-world shopfloor scenario. Therefore, care was taken to 
have different assembly tasks such as screwing, fastening, circlip fixing, and 
bearing fitting. An exploded view of the planetary gear is shown in Figure 8.4. All 
the parts are 3D printed, and components such as bearings, circlips, nuts, and bolts 
are bought off the shelf for the assembly operation. The usage of several tools is 
required by including such components, which increases the complexity of the 
assembly process and simulates the shopfloor experience. In Sections 8.4.2 and 
8.4.3, two AR applications are developed to digitalize the work instruction to 
assemble a planetary gearbox. 
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FIGURE 8.4 Exploded view of planetary gear. 
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8.4.2 Case Stupy 1 — DIGITALIZING ASSEMBLY INSTRUCTIONS USING A 
HANDHELD AR DEVICE 


This section presents the development process of digitalizing the assembly instruc- 
tions to assemble a planetary gear using an HHD (e.g., a mobile phone). From the 
hardware perspective, a Samsung® A7 device running on the Android operating 
system is chosen. The device comes with all the necessary AR elements, such as a 
camera, processing unit, touch interface, and other useful sensors. Moreover, mobile 
phones are readily available, and most users feel comfortable using these devices. 
From a software perspective, a framework is developed to model the abstraction 
first approach. As the manufacturing domain is adapting software to assist their daily 
operations, it is important to understand and adapt the workflow of software devel- 
opers. In software development, the C4 framework (Brown, 2019) is very popular for 
visualizing the software elements in a simple using a lean graphical representation 
technique. Figure 8.5 illustrates the software framework used to design the Android 
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FIGURE 8.5 Software framework for mobile device (HHD). 
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AR application on a mobile device. The C4 framework has three layers: software 
system, containers, and components. The idea of a user interacting with the AR 
application forms the software system. Containers are executable applications, and in 
developing the Android application, the following containers are used: Vuforia®, 
Unity”, and Lean Touch. Components provide functionality, and are not separately 
deployable without a well-designed interface. In this application, C# programming 
and 3D files are examples of components. 

The virtual instructions to assemble the planetary gear are shown using anima- 
tions created by 3D models. The user can use the interactive buttons on the UI to 
reproduce, pause, replay, and navigate to the next/ previous instructions. The UI is 
designed such that the user can get auxiliary information about the assembly parts 
by tapping each part. The auxiliary information is available on demand, as having it 
on the UI by default is not ideal for the usability of the application. The user also 
can scale and rotate the animations by pinching the display unit with two fingers. 
Touch input modality was applied in the development of this AR application. The 
program is developed from the Unity® scenes containing all the necessary elements 
to run on it. AR Software Development Kit (SDK) from PTC (PTC, 2020) is used to 
convert the Unity® scenes to an executable Android application using an inbuilt 
plugin. In the development of the application, a fiducial marker (QR code) opti- 
mized by Vuforia® is used to track image target and orientation. Figure 8.6 illus- 
trates the UI of the Android application. 

In summary, the AR application is developed for Android devices using see- 
through video technology using a marker-based tracking system. Touch modality is the 
only input modality used for the development of this AR application. The application is 
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FIGURE 8.6 User Interface (UT) on mobile device (HHD). 
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portable to different locations on the shop floor. However, the FoV of the UI is limited 
by the size of the display unit of mobile devices. Visual cues such as part and tool 
pointers are not included in the application as it requires much more computing power 
and creates lags while running the application. 


8.4.3 Case Stupy 2 — DIGITALIZING ASSEMBLY INSTRUCTIONS USING 
A HeaD-MounteD AR DEVICE 


Like the HHD application in Section 8.4.2, the application on HMD (e.g., 
*HoloLens) delivers the digital instructions to assemble a planetary gear through 
several steps. The hardware used in this development process is an HMD from 
Microsoft® running on the Windows® operating system. HoloLens® 1 is packed with 
several sensors and a processing unit and supports input modalities such as gaze 
tracking, gesture recognition, and voice recognition. The FoV in HMD devices can 
be varied by head movement. This new freedom allows the implementation of 
several new features compared to the HHD application to help the user navigate the 
application. However, the user needs to be trained before using the HoloLens® as it 
is relatively new and unfamiliar to most users. 

From a software perspective, a framework is developed just like in case study 1. 
Figure 8.7 illustrates the software framework used to design the Android AR 
application on a HoloLens®. The three layers of the C4 framework represented here 
are the software system (HoloLens® application), containers (Unity and Mixed 
Reality Tool Kit), and components. The spatial mapping feature of HoloLens® is 
used to detect the workplace and display instructions instead of a QR code. The 
voice commands from the user are translated to inputs for the HoloLens®, and this is 
enabled by the inbuilt Voice Manager feature of the Mixed Reality Tool Kit. 

When the user interacts with the HoloLens® for assembly instructions, by 
default, a scene is loaded without any supplementing information. This will keep 
the visual and cognitive load of the user to a minimum level. However, the user can 
command the device using voice inputs such as “Parts,” “Tools,” “Next,” and 
“Previous” to get the information needed or to navigate through different scenes. 
The hand gesture is also enabled in the application and is limited to positioning the 
digital workstation on top of the real one. This is done to situate the origin of the 
digital system on the correct location of the spatial map; after that, it is anchored 
and will stay locked in position even when the application is restarted. This initial 
setup is done when the application is used for the first time or when the workstation 
is physically moved somewhere other than the initial place. This also enables the 
device’s portability for different workstations for diverse operations. Figure 8.8 
illustrates the UI of the application on HoloLens?. 

In summary, the AR application is developed for HoloLens® using see-through 
optical technology using spatial mapping as a tracking system. Speech and gesture 
modalities are incorporated into the application, validating the multimodal approach 
in interacting with AR devices. The application is portable to different locations on 
the shop floor. Moreover, the FoV of the UI is variable enabled by the head tracking 
feature of the device. Visual cues such as part and tool pointers are included in the 
application as an increased field of view allows multiple feature implementations. 
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FIGURE 8.7 Software framework for HoloLens* (HMD). 


8.5 VR TECHNOLOGY IN MANUFACTURING 


The availability of powerful, immersive hardware and software systems has made 
the visualization of complex systems in a realistic virtual environment very effi- 
cient. VR is defined as an immersive computing technology that provides a unique 
way to interact with the ever-growing digital landscape (Berg et al., 2017). This 
interaction helps in depicting the behavior of complex and abstract systems in a 
holistic way to both experts and non-experts. The ability of VR technology to 
revolutionize traditional processes and working methods has been identified by 
large companies in the automotive and aircraft sector, especially in the topics of 
knowledge transfer, professional training, and new product development. 

The objective of the immersive technology tools developed to experience the 
virtual environment is mainly to deliver information to our senses, particularly 
sight, hearing, and touch. Some of the most used VR tools are VR-Powerwall, cave 
virtual environment, and a VR-head-mounted display. 
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The rest of the sections will focus on the important applications of VR in 
manufacturing and the various case studies which have shown the potential of VR 
in manufacturing. 


8.6 VR APPLICATIONS IN MANUFACTURING 


VR is being used increasingly in many manufacturing applications. The various 
applications are explained in this section to provide the reader with the possible 
opportunities for using VR (Choi et al., 2015). 


8.6.1 VR IN DESIGN 


VR technology plays a significant role in designing and prototyping new products. 
The best use of VR is in the conceptual design stage, where it provides the designers 
with a virtual environment for designing a new product in 3D. This feature helps 
them in analyzing the performance of different mechanical features such as hinges, 
assembly, etc., which in turn will aid in evaluating the conceptual design. 

Another important application of VR is seen in virtual prototyping, where the 
engineers can use the virtual prototype before fabricating the physical prototype to 
suggest improvements in the design, evaluate specific characteristics of the proto- 
type, help in manufacturing planning, and get feedback from the potential cus- 
tomers in case of a new product. Generally, such applications incorporate all the 
different functionalities needed to realistically simulate the dynamic behavior of 
the product, including human interaction with the virtual environment, and consist 
of both offline and real-time virtual simulation. 
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8.60.2 VR IN MANUFACTURING PROCESSES 


VR technology helps in understanding three different areas of the manufacturing 
process, namely, machining, assembly, and inspection. Machining using VR 
technology is used in processes such as turning, milling, drilling, grinding, and 
so on. They help the engineers in simulating the manufacturing processes where 
the user can mount the workpiece on the machine, perform direct machining 
operations using the corresponding tool and verify if the desired output is 
obtained. 

Virtual assembly using VR technologies helps in making assembly-related en- 
gineering decisions as it helps the designers to analyze, visualize and implement 
different models without realization of the product or support process. On a larger 
scale, it can help in investigating the assembly process in an assembly line where 
the location, sequence, and type of operation can be optimized beforehand. 


8.6.3 VR IN PLANNING, SIMULATION, AND TRAINING 


The virtual environment developed using different VR tools can lead to optimal 
planning of a manufacturing system with the best possible arrangement of shop floor 
and different machines. The comparison of different layouts/solutions considering 
human experiences and facts has resulted in the rapid start of several manufacturing 
processes. 

VR-based training is among the most advanced method for training workers with 
the necessary manufacturing skills and processes. The virtual model of the product, 
plant layout, and assembly process helps the workers understand the different 
features associated with each field. Since the visualization and simulation are done 
realistically, the workers feel it is easier and faster to learn than with the conven- 
tional methods. 


8.7 VR CASE STUDIES 


8.7.1 Case Stupy 1 — VR FOR THE DEVELOPMENT OF A VIRTUAL ROBOTIC CELL 


Darmoul et al. (2015) developed a virtual model of a robotic cell and analyzed 
its functioning in a semi-immersive virtual environment to provide a feasible 
layout planning for setting up the real robotic cell. Through this work, the authors 
demonstrated that saving time and effort while designing such robotic cells using 
VR technology was possible. They were able to assess various layouts and 
configurations without the risk of damaging the system. They used the virtual 
robotic cell for layout planning and set up a real robotic cell in an advanced 
manufacturing institute, King Saud University, Saudi Arabia. 

To achieve their goals, they used a combination of hardware equipment and 
software architecture consisting of CATIA for the CAD model generation, 
InterSense IS-900 wired tracking devices for head and hand tracking, pro-engineer, 
division mockup, and so on. The developed robotic cell was used for training 
industrial workers and academic researchers to demonstrate the advantages of VR 
technology and its potential for the manufacturing industry. 
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8.7.2 Case Struny 2 — VR-BASED CONFIGURATOR OF INTERIOR ACCESSORIES 
FOR FIRE TRUCK 


Researchers from the University of Applied Sciences Karlsruhe, Germany, in col- 
laboration with company Rosenbauer Karlsruhe GmbH & Co. KG, used VR tech- 
nology to develop a configurator of interior accessories for a fire truck (Bellalouna, 
2020). This helped the company decouple the determination of the firefighting 
equipment position from the production of the fire truck. The author identified the 
necessary functions to be implemented in the VR application by organizing several 
workshops with the sales engineering departments of the company and different fire 
departments. Some of the necessary functions were VR libraries containing all the 
firefighting accessories, different fire truck models, the configuration of the equipment 
shed, a function to determine the equipment weight, center of gravity, and axle load, 
and finally, an option to create the bill of materials. The VR application developed 
was evaluated to check its usability for further business processes, and the following 
points were inferred: 


a. The users found the functions to be intuitive and also allowed a cognitive 
configuration process. 

b. The application helped in creating a realistic VR environment with a high 
emersion ability without any restriction. 

c. The application helped in identifying conflicts between design and cus- 
tomer requirements during the early product lifecycle phase. 

d. One weak point of the system was that the successful VR-based config- 
urator only allowed a single-user application. In contrast, successful 
deployment of the VR application requires multi-user involvement. 


8.7.3 Case STUDY — IMPLICATIONS OF VR TECHNOLOGY ON ENVIRONMENTAL 
SUSTAINABILITY IN MANUFACTURING INDUSTRIES 


An interesting case study was carried out by Chen et al. (2021) with an international 
automotive company with a research and development center in Sweden and man- 
ufacturing plants in China to explore the implications of VR technology on en- 
vironmental sustainability in manufacturing industries. They developed a VR demo of 
the welding process in a manufacturing environment in China to be used by the users 
from the development center in Sweden. In addition to the VR demo, they used 
various tools such as customer experience feedback, interviews, and questionnaires. 

They showed that the users were able to communicate more efficiently about the 
product design as the VR application offered them the product view from different 
angles and with more details. This improved communication efficiency enabled 
constant and frequent dialogue between developers and implementation sites and 
potentially reduced the number of travels between China and Sweden. 

Due to the possibility of performing product and process design verification from 
Sweden, this demo application could contribute to environmental sustainability with 
less CO, emissions generated from international trips. 
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FIGURE 8.9 Conceptual framework for integrating MR technology in a manufacturing plant. 


8.8 CONCEPTUAL FRAMEWORK FOR MR TECHNOLOGIES 


MR, like several other Industry 4.0 solutions, draws in data from the tools available 
to create virtual realities the workers can use to augment their work. Implementing 
MR technologies by using VR and AR for both on-site and off-site operations in a 
manufacturing sector can help improve the working conditions for the workers, 
have better control over the plant operations, and integrate Industry 4.0 technologies 
for their production activities. A simplified conceptual framework for MR tech- 
nology in manufacturing is shown in Figure 8.9. 


8.9 DISCUSSION AND CONCLUSION 


MR technologies play a very significant role in complex product development and 
production processes. Due to the advancement in different immersive technologies, 
MR has found its way from many high-tech industry sectors to SMEs as well. 
However, there are several challenges that need to be addressed to reach its actual 
potential in the field of manufacturing. 

One of the topics that will strongly influence the future of MR applications 
development is spatial registration technology, which can combine the virtual world 
and the real world through a proper relationship of relative positions. This will help 
the user in efficient localization in the MR environment and improve the performance 
of the MR application. The second challenge is the development of a user-friendly 
interface for MR applications that are affected by aesthetics and ergonomics. One big 
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problem in using MR technology at a large scale is the heavy display equipment, 
which is unsuitable for a production line. The alternative solution of using mobiles or 
smartphones also has a limitation: the workers must perform their work simulta- 
neously using the MR application. During such times, fixing the mobile phone in the 
workplace or using a lightweight advanced HMD could be a good option. The next 
challenge is in developing MR applications for multiuser collaboration. Generally, a 
fully functional manufacturing line consists of a line manager, engineers from dif- 
ferent disciplines, operators, and technicians. In such cases, it is of utmost importance 
to provide an MR platform enabling helpful collaboration, information sharing, and 
real-time communication among the different users. 

Despite their limitations, the potential of MR technology for manufacturing is 
enormous and is becoming increasingly mature through the innovation of related 
technologies. Combining the benefits of mixed Reality with other Industry 4.0 tools 
will enable the manufacturing sector to bring different teams together to increase 
their performance, efficiency, and market share. 
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